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REGRESSION, FACTOR ANALYSIS, AND CROSS-VALIDATING PROCEDURES 
WERE USED TO ASSESS THE IMMEDIATE AND DELAYED LEARNING 
EFFECTS OF PROGRAMMED INSTRUCTION. INDEPENDENT VARIABLES 
STUDIED WERE IN THE AREAS OF VERBAL LEARNING, READABILITY, 
INCIDENTAL LEARNING, AND RESPONSE DIFFICULTY. 144 PAID 
STUDENT VOLUNTEERS, PARTICIPANTS IN A PRIOR STUDY, WERE 
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INCLUDE— CATEGORIES OF LEARNING, ESTIMATE OF THE EXTENT TO 
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FOREWORD 

The study reported here is one of several related stud- 
ies recently completed or nearing completion within the 
Instructional Media Research Unit, Purdue, These recent 
studies include the M.S. thesis of the present senior author, 
the dissertation of Jon B. Myers, and the nearly completed 
dissertation of Robert W, Heckman, The present study, plus 
the senior author's thesis and the Myers dissertation, employ 
correlational and multiple regression analysis in searching 
for those stimulus variables within verbal instructional 
material which are predictive of observed instructional 
effects. Typically, their dependent variables are indexes 
of learning gain that are associated with individual criterion 
items. 

The first of these studies, the Myers dissertation, was 
generally successful in its use of quantified stimulus vari- 
ables to predict posttest performances, employing data from 
both an "immediately" administered criterion measure and 
from the same measure administered (to a different randomly 
constituted group) following about one month of post-instruc- 
tion delay; multiple correlations ranged from the mid-fifties 
to the mid-sixties. The present senior author's thesis 
seemed even more successful in its search for stimulus vari- 
ables which could be combined to predict instructional 




effects. In that study, the subject matter was spelling and 
the multiple correlations ranged from the seventies to the 
mid-eighties . 

Without additional work, however, it would have been 
unwise, to say the least, to emphasize the substantive 
results of these earlier studies. Each had its share of pro- 
cedural problems and these problems required early considera- 
tion and, where possible, remedy. It was known, for example, 
that the studies incorporated more stimulus (predictor) vari- 
ables than was desirable. Because of this, impressive levels 
of prediction might be initially developed, but might also 
fail in cross-validation. An important need was therefore 
to provide immediately for cross-validation of results, if 
not also to restrict the number of variables included. In 
the present study, cross-validation and some reduction in 
the number of variables have been provided. 

There was a further difficulty stemming from the pos- 
sible identification of stimulus variables which, while not 
causing” the effects with which they correlated, were never- 
theless predictive of effects, perhaps because of their cor- 
relation with obscured causal variables. This problem sug- 
gests the use of factor analytic methods as one means of 
reducing the many stimulus variables to a smaller set of 
principal stimulus dimensions. This, too, has been provided 
for within the present study; the stimulus matrix was factor 
analyzed and the orthogonal factor 



scores employed to predict 
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the learning criteria. 

Still another interest and concern stems from the pos- 
sible existence of "aptitude-by-treatment" interactions. If 
present, they could contribute to the finding of relation- 
ships which would change as other changes were made, either 
in relevant aptitude characteristics of the learner group or 
in characteristics of the stimulus material. Although apti- 
tude-by- treatment interactions have not been directly consid- 
ered in the present study, they are the concern of the 
Heckman dissertation and are being investigated with many of 
the same data employed in the present study. 

There are continuing problems also in the identifica- 
tion of the stimulus material that constitute an instruc- 
tional "sequence" (i.e., the set of sentences, typically, 
which contribute to development or shaping of each criterion 
behavior) j when this cannot be resolved, even the otherwise 
simple matter of determining frame count and sentence count 
becomes unreliable. Once a sequence is determined, further 
problems remain. In sets of hundreds or even thousands of 
sentences, as in the present study, it is no simple matter 
to identify accurately every appearance of concrete referent 
nouns, affixed words, prepositional phrases, and sequence 
prompts, to name just a few variables which may be sources 
of difficulty. For the present study, it is suspected that 
problems in the correct identification of sequence sentences 
and of indexing their contents are more acute than in the 
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related and earlier studies. The senior author is endeavor- 
ing to re-exainine the instructional material used in the 
study and to improve its analysis-indexing. 

Since much of the prediction initially derived from pre- 
dictor variables in this study failed in cross-validation, 
it might be argued that difficulties inherent in the proced- 
ures are too great and that it is time to return to more cus- 
tomary study designs. We do not see such argument as con- 
vincing and believe still that there are reasons for continu- 
ation of the work. First, the present efforts at prediction 
and the identification of functioning stimulus variables 
were not totally a failure; some prediction remained in 
cross-validation. Second, alternative designs for the study 
of functioning variables within complex stimulus fields typi- 
cally make their own compromises and have their own problems; 
to shift over would not provide pure gain. Finally, there 
is the continuing appearance of speed and economy in the 
present procedures, if these can be suitably refined. This 
view is partially quoted on pages Z and 1 of the present 
report. More completely, the quotation (from Donald Campbell 



and Julian Stanley, as cited) is 



uch correlational data are relevant to causal hypo- 
theses inasmuch as they expose them to disconf irmation 
If a sero correlation is obtained, the credibilitv 
of the hypothesis is lessened. If a high correla- 
tion occurs, the credibility of the hypothesis is 
strengthened in that it has survived a chance of 
disconfirmation. To put the matter another way 
correlation does not necessarily indicate causation 
but a causal law of the type producing mean differ-* 
ences in experiments does imply correlation ... the 




relatively inexpensive correlational approach can 
provide a preliminary survey of hypotheses, and those 
which survive this can be checked through ths more 
expensive experimental manipulation. 

The problem and the procedures now seem clearer, even 
though most of the answers are not. 
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INTRODUCTION 

This study was designed to identify characteristics of 
linear programmed instruction which a»e predictive of immedi- 
ate and delayed learning effects. The study was sponsored 
by the Instructional Media Research Unit as part of its 
efforts to explore the relationships between and interac- 
tions among characteristics of instructional material and 
learners. The objective is a system in which instructors 
may identify "relevant” characteristics of an audience, then 
provide an instructional program incorporating characteris- 
tics to optimize learning. 

The present study was designed to overcome certain crit- 
icisms of the published research on programmed instruction. 
One disquieting trend is the inattention to the complexities 
of instructional stimuli. The typical experimental design 
entails manipulation of one or two independent variables 
while presumably holding constant the multitude of other var- 
iables which may influence learning, When used for instruc- 
tional research, this strategy appears deficient in several 
respects. First, since no comprehensive theory yet exists 
to guide research, experimenters too often have little basis 
for selecting variables to be manipulated. Secondly assum- 
ing that important variables are chosen, the experimenter 
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must still select "dosage" levels within a crucial range. 
Finally, in many studies, control of variables not under 
investigation is more apparent than real. In fact, the mul- 
titude of variables which affect learning may virtually pre- 
clude the degree of control needed when manipulating indepen- 
dent variables. Illustrations of inadequate controls are 
numerous. Lumsdaine ( 1963 , p. 627), for example, says the 
following about two studies of step site (Coulson and 
Silberman, 1959; Evans, Glaser, and Homme, 1959): "smaller 
and more numerous steps gave better learning scores, but 
these results. . .are difficult to interpret because the small 
step program (1) provided more practice on varied examples, 
and (2) took considerably more time." 

The point is not that designs which consider only a 
very few variables are inappropriate. Rather, it is con- 
tended that the means should be developed for measuring, or 
otherwise more exactly specifying, the variables in program- 
med materials. . (Holland, 1965, p. 92). It is believed 
that a multiple regression design may provide some of these 
"means." Wot only does regression analysis yield quantita- 
tive statements of variables and their relationships, but it 
also readily accommodates virtually an unlimited number of 
independent variables, which yields additional advantages. 
First, if one views instructional research as still in an 
exploratory stage, then "...the relatively inexpensive corre- 
lational approach can provide a preliminary survey of 
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hypotheses, and those which survive can be checked through 
the more expen si re experimental manipulation" (Campbell and 
Stanley, 1 963, p. 234 ) . Secondly, multiple regression analy- 
sis minimises the immediate need te control stimulus vari- 
ables experimentally. Statistical controls inherent in such 
analyses provide at least a partial substitute. Thirdly, 
regression analysis can be extended to include factor analy- 
sis as a means of arranging stimulus variables into a parsi- 
monious set of underlying factors, finally, cross-valida- 
tion designs may be employed to provide safeguards against 
premature identification of insignificant stimulus-effects 
relationships • 

Programmed instruction research can also be criticized 
for the degree to which a few program variables dominate the 
literature to the exclusion of other promising variables. 

For example, the field of verbal learning has lately pro- 
duced findings which suggest the importance of syntactic 
(i.e., sentence) and semantic (i,e. t word) variables in ver- 
bal learning. While these developments are based largely on 
artificial laboratory tasks, they nevertheless appear rele- 
vant to programmed instruction. Several investigators have 
shown grammatical structure to be a significant factor in 
rote learning (Bpstein, 1961, 1962; Coleman, 19655 Harks and 
Miller, 1964 ) and in paired associate learning (Glanzer, 

1962; Martin, Davidson, and Williams, 1965). In consenting 
on such results, Epstein ( 1961 ) and Miller (1962) believe 
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that syntax organises words into efficient chunks for memor- 
ization, As Miller (1965) later wrote: "the syntactic struc- 
ture of a sentence imposes groupings that govern the inter- 
actions between the meanings of the words in that sentence" 
(p, 18), Other experimenters have dealt with the contextual 
constraint effects of syntax. The problem investigated in 
such research is: given a sentence with a word deleted, does 
gram... tical structure aid a subject in guessing the missing 
word. Aborn, Rubensteln and Sterling (1957) found that "the 
length, distribution, and grammatical structure of... (a sen- 
tence)... are all Independently effect. * sources of con- 
straint on words in sentences." Within the field of program- 
med instruction Markle (1964, p. 57 ) has described syntax as 
a "formal" prompting device. 

Additional evidence for the role of syntax in written 
verbal instruction comes from studies of specific grammatical 
structures. Gray and Leary (1935), for example, report data 
indicating that simple sentences may be easier to comprehend 
than either compound or complex sentences. In a study of 
grammatical transformations and their relation to rote learn- 
ing of sentences, Coleman (1965) concluded that two short co- 
ordinate clauses tended to be retained better than an equiva- 
lent long clause. "Apparently ?• person can process content 
morphemes packaged into two clauses more easily than he can 
process content morphemes packaged into a single clause" (pp. 
340-341). Within a programmed Instruction setting. Smith 
(1965) analyzed the relation of learning effects with 
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certain grammatical structures which he called "multi-worJ 
modifiers, •»§. • prepositional phrases, verbal nouns and 
adjeotives, For a sample of young children, these variables 
appeared detrimental to learning. Partial support for this 
finding comes from Gray and Leary (1935) who report a nega- 
tive correlation between prepositional phrases and reading 
comprehension but a positive correlation between infinitive 
phrases and comprehension. Miller (1951, p. 134) prudently 
notes that effects of sentence complexity may be confused 
with sentence length effects. 

Along with syntactic factors, Marks and Miller (1964) 
noted that semantic factors were related to errors in memor- 
sentences. For example, Myers (1964) in a programmed 
learning study and O'Connor (1950) in a film learning study 
both observed a relationship between frequency of affixes 
and learning effects. Smith { 1965 ) also found a positive 
correlation between suffixes and learning gains, Yngve 
( 1962 ) speculated that affixes reduce sentence complexity by 
replacing other words. On the other hand, Miller ( 1951 , p. 
134) theorized that "counting affixes is another way of esti- 
mating the proportion of hard words." Since affixes increase 
the average of syllables per word, it is natural ic suppose 
a relationship between syllables and learning. Smith ( 1965 ) 
and Gray and Leary (1935), however, report conflicting 
findings. The former obtained a slight positive correla- 
tion between syllables per word and learning while 
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the latter observed a negative correlation. Lastly, Smith 
(1965) also analysed the effect of modifiers (adjectives, 
adverbs and possessive pronouns) and concluded that they 
facilitate learning of written instruction. 

The literature on readability furnishes additional sup- 
port for the notion that syntactic and semantic variables 
influence learning. The Flesch and most other readability 
formulae consist of a word variable (e,g«, number of syl- 
lables per 100 words) and a sentence variable (e,g«, number 
of words per sentence). In reviewing studies of reading com- 
prehension as a function of reading ease, Klare ( 1963 , pp. 
13^-135) concluded that ", . .variation of one factor (word or 
sentence ) was not sufficient to provide increased comprehen- 
sion, but that a combination of the two was,” Comparative 
studies of readability measures suggest that close prv cedure 
may be the most sensitive measure of reading ease, (See, 
for example, Taylor, 1953). Greene ( 1965 ) prefers to think 
of close procedure as "a means of measuring the degree of 
correspondence between the language of a message and the lan- 
guage system of a reader 11 (p. 213). Another characteristic 
which makes close procedure attractive for programmed instruc- 
tion research is its similarity to the experimental task of 
textual constraint studies. Furthermore, Weaver and 
Kingston ( 1963 ) in their factor analytic study of reading 
and language ability have shown that close procedure is 
closely correlated with a factor they call redundancy 
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retrieval, "The underlying ability here seems to be the rec- 
ognition of redundancy characteristics of language" (p. 259 ) , 
a property one might hypothesize as related to the comprehen- 
sion of programmed materials. 

In addition to syntactic and semantic variables, other 
factors which bear investigation include incidental learning 
and response difficulty. The incidental presentation of 
responses to be learned is so common, one wonders about its 
role in programmed instruction. Likewise, little attention 
has been given to differences in the difficulty of the 
responses to be learned. Prior studies of the Instructional 
Media Research Unit have assumed for convenience that the 
responses to be learned were of equivalent difficulty. It 
is here hypothesized that the length of the response to be 
learned may provide a quantitative measure of the difficulty 
of the learning task. 

Finally, many pay lip-service to assessing retention 
effects of instruction, but few do anything about it. An 
instructional treatment may produce relatively inefficient 
learning but greater resistance to forgetting than other 
methods. A prominent example is the use of variable- ratio 
reinforcement schedules in animal studies. Within program- 
med instruction, Krumboltz and Weisman (1962) compared the 
effects of overt and covert responding and found that "writ- 
ten responses to programmed materials were found to aid in 
the retention of new learning when measured after two weeks 
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interval although an immediate test of learning failed to 
show any differences.” 

fo summarize, this study employed a multiple regression 
design to assess the immediate and delayed learning effects 
associated with a number of characteristics of programmed 
instruction. The characteristics selected for study include 
variables previously studied within the context of programmed 
instruction and variables recently investigated in the areas 
of verbal learning and readability. Tie effects of inciden- 
tal learning and of response difficulty were also selected 
for study. 
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PROCEDURE 

Subjects 

One hundred forty-four Purdue students who had partici- 
pated in a prior study of the Instructional Media Research 
Unit served as Ss; they were paid volunteers recruited ori- 
ginally from freshman English courses. The Ss were divided 
into three groups: the experimental group (65 Ss), the con- 
trol group (60 Ss), and the cloze score group (19 S s ) . The 
experimental and cloze score groups were randomly selected 
from the randomly constituted experimental group of the 
prior study. Similarly, control £s were drawn from the ear- 
lier control group. Through the course of the study four 
students dropped from the experimental group and five from 
the cloze score group; the study's analyses are thus based 
on the remaining Ns of 6l, 60, and lU. 

The median age for all groups was 18 years, end the 
sexes were distributed as follows: 31 males and 30 females 
in the experimental group, 35 males and 25 females in the 
control group, and five males and nine females in the cloze 
score group. More complete descriptions of the Ss are 
reported by Fincke (1967). 
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